Road condition deep learning model

ABSTRACT

The technology relates to using on-board sensor data, off-board information and a deep learning model to classify road wetness and/or to perform a regression analysis on road wetness based on a set of input information. Such information includes on-board and/or off-board signals obtained from one or more sources including on-board perception sensors, other on-board modules, external weather measurement, external weather services, etc. The ground truth includes measurements of water film thickness and/or ice coverage on road surfaces. The ground truth, on-board and off-board signals are used to build the model. The constructed model can be deployed in autonomous vehicles for classifying/regressing the road wetness with on-board and/or off-board signals as the input, without referring to the ground truth. The model can be applied in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.16/893,664, filed Jun. 5, 2020, the entire disclosure of which isincorporated herein by reference.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or cargo fromone location to another. Such vehicles may operate in a fully autonomousmode or a partially autonomous mode where a person may provide somedriving input. In order to operate in an autonomous mode, the vehiclemay employ various on-board sensors to detect features of the externalenvironment, and use received sensor information to perform variousdriving operations. Road conditions including water on the roadway mayadversely impact operation of the vehicle, including how informationfrom the sensor system is evaluated, when a wiper system is engaged,real-time and planned driving behavior, among other issues.

BRIEF SUMMARY

The technology relates to using on-board sensor signals, otherenvironmental information and a deep learning (DL) model to classifyroad wetness and/or to perform a regression analysis on road wetnessbased on a set of input information. The input information may includeon-board and/or off-board signals obtained from one or more sources suchas on-board perception sensors mounted along the vehicle, other vehicleon-board modules, external weather measurement, external weatherservices, etc. Ground truth includes measurements of water thickness onroad surfaces, such as water film thickness and/or ice coverage. Theground truth, on-board signals from sensors or systems of the vehicle,and off-board signals from sources other than the particular vehicle,are used to build a DL model. The constructed model can be deployed inautonomous vehicles for classifying/regressing the road wetness withon-board and/or off-board signals as the input, without referring to orotherwise relying on the ground truth during real-world autonomousdriving. The model can be applied in a variety of ways to enhanceautonomous vehicle operation, for instance by altering current drivingactions, modifying planned routes or trajectories, activating on-boardcleaning systems, etc.

According to one aspect, a method for generating a road condition deeplearning model is provided. The method comprises receiving as a firstset of training inputs, by one or more processors, sensor data of anenvironment along a portion of a roadway from one or more on-boardvehicle sensors; receiving as a second set of training inputs, by theone or more processors, off-board information associated with theportion of the roadway; evaluating, by the one or more processors, thereceived first set of training inputs and the received second set oftraining inputs with respect to ground truth data for the portion of theroadway, the ground truth data including one or more measurements ofwater thickness across one or more areas of the portion of the roadwayto give classification or continuous estimation of wetness along the oneor more areas of the portion of the roadway, wherein the evaluatinggenerates road wetness information based on the received first andsecond sets of training inputs and the ground truth data; generating theroad condition deep learning model from the road wetness information;and storing the generated road condition deep learning model in memory.

The received sensor data may include one or more of lidar returns,camera still images, camera video, radar returns, audio signals, oroutput from a vehicle on-board module. The off-board information mayinclude one or more of weather station information, public weatherforecasts, road graph data, crowdsourced information, or observationsfrom one or more other vehicles.

The method may further comprise performing signal fusion of some or allof the received sensor data and the received off-board information. Themethod may also include applying weighting to different signals of thereceived sensor data. In an example, the method further includes, priorto construction of the road condition deep learning model, performing astatistical analysis to determine which received sensor data correlateswith the ground truth data. Here, the method may also includedeemphasizing any received sensor data that does not meet a correlationthreshold with the ground truth data.

The method may further include, prior to construction of the roadcondition deep learning model, masking out information of one or moredynamic objects on the roadway from the received sensor data. The methodmay include, prior to construction of the road condition deep learningmodel, limiting the received sensor data to a selected range or distancefrom the one or more vehicle sensors. The method may include smoothingground truth measurements of the ground truth data prior to constructionof the road condition deep learning model.

In an example, the method further includes applying the road conditiondeep learning model to one or more roadway regions to either identify aprobability of wetness for each of the regions or estimate water filmdepth for each of the region. In another example, the method includescausing one or more systems of a self-driving vehicle to perform atleast one of altering a current driving action, modifying a plannedroute or trajectory, or activating an on-board cleaning system of theself-driving vehicle based on the road condition deep learning modeloutput.

In a further example, storing the generated road condition deep learningmodel in memory comprises storing the generated road condition deeplearning model in memory of one or more vehicles that are not equippedwith a ground truth measurement sensor. The stored road condition deeplearning model is configured for use in evaluating real-time roadwetness based on real-time sensor data and selected off-boardinformation.

The ground truth data may include human-labeled road wetness examples.And the one or more measurements of water thickness may be one or moremeasurements of water film thickness or ice coverage across the one ormore areas of the portion of the roadway.

According to another aspect, a system is configured to operate a vehiclein an autonomous driving mode. The system includes memory storing a roadcondition deep learning model. The model relates to a discreteclassification or continuous regression/estimation of road wetness. Thesystem also includes one or more processors operatively coupled to thememory. The one or more processors are configured to receive sensor datafrom one or more sensors of a perception system of the vehicle whileoperating in the autonomous driving mode. The one or more sensors areconfigured to detect objects or conditions in an environment around thevehicle. The one or more processors are configured to use the storedmodel to generate information associated with the discreteclassification or continuous regression/estimation of road wetness basedon the received sensor data, and to use the generated information tocontrol operation of the vehicle in the autonomous driving mode.

In one example, the model is formed by evaluating a first set oftraining inputs of sensor data of an environment along a portion of aroadway from one or more on-board sensors and a second set of traininginputs of off-board information associated with the portion of theroadway with respect to ground truth data for the portion of theroadway. The ground truth data includes one or more measurements ofwater thickness across one or more areas of the portion of the roadway.The first set of training inputs of sensor data may include one or moreof lidar returns, camera still images, camera video, radar returns,audio signals, or output from a vehicle on-board module. The second setof training inputs of off-board information may include one or more ofweather station information, public weather forecasts, road graph data,crowdsourced information, or observations from one or more othervehicles.

Controlling operation of the vehicle in the autonomous mode using thegenerating information may include at least one of alteration of acurrent driving action, modification of a planned route or trajectory,or activation of an on-board cleaning system.

According to another aspect, a vehicle is able to operate in anautonomous driving mode, in which the vehicle includes the systemconfigured to operate the vehicle as well as the on-board perceptionsystem as discussed above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-B illustrate an example passenger-type vehicle configured foruse with aspects of the technology.

FIGS. 1C-D illustrate an example cargo-type vehicle configured for usewith aspects of the technology.

FIG. 2 is a block diagram of systems of an example passenger-typevehicle in accordance with aspects of the technology.

FIGS. 3A-B are block diagrams of systems of an example cargo-typevehicle in accordance with aspects of the technology.

FIG. 4 illustrates an example of detecting ground truth in accordancewith aspects of the technology.

FIG. 5 illustrates example sensor fields of view for a passenger-typevehicle in accordance with aspects of the disclosure.

FIGS. 6A-B illustrate example sensor fields of view for a cargo-typevehicle in accordance with aspects of the disclosure.

FIG. 7 illustrates an example system in accordance with aspects of thetechnology.

FIG. 8 illustrates examples of on-board and off-board training inputs inaccordance with aspects of the technology.

FIG. 9 illustrates an example of roadway wetness in accordance withaspects of the technology.

FIG. 10 illustrates an example road condition deep learning model inaccordance with aspects of the technology.

FIGS. 11A-B illustrate driving modification scenarios in accordance withaspects of the technology.

FIGS. 12A-B illustrate an example system in accordance with aspects ofthe technology.

FIG. 13 illustrates an example process in accordance with aspects of thetechnology.

DETAILED DESCRIPTION

As noted above, aspects of the technology use ground truth informationabout road wetness, input from one or more of the following sources suchas other on-board sensor signals and/or signals from other on-boardmodules, and off-board signals to develop a DL model for road wetnessclassification, as well as to perform a road wetness regressionanalysis. Certain data (e.g., on-board sensors and off-board signals) isused in the DL model, while other data (e.g., ground-truth info) mayonly be used only for training. Thus, a deployed system does not requirethat the ground-truth sensors be installed on the vehicle. For instance,training inputs are evaluated with respect to ground truth informationfor a given roadway segment. The output of the DL model can be used in avariety of ways to enhance autonomous vehicle operation, for instance byaltering current driving actions, modifying planned routes ortrajectories, activating on-board cleaning systems, etc.

Example Vehicle Systems

FIG. 1A illustrates a perspective view of an example passenger vehicle100, such as a minivan, sport utility vehicle (SUV) or other vehicle.FIG. 1B illustrates a top-down view of the passenger vehicle 100. Thepassenger vehicle 100 may include various sensors for obtaininginformation about the vehicle's external environment. For instance, aroof-top housing 102 may include a lidar sensor as well as variouscameras, radar units, infrared and/or acoustical sensors. Housing 104,located at the front end of vehicle 100, and housings 106 a, 106 b onthe driver's and passenger's sides of the vehicle may each incorporateLidar, radar, camera and/or other sensors. For example, housing 106 amay be located in front of the driver's side door along a quarter panelof the vehicle. As shown, the passenger vehicle 100 also includeshousings 108 a, 108 b for radar units, lidar and/or cameras also locatedtowards the rear roof portion of the vehicle. Additional lidar, radarunits and/or cameras (not shown) may be located at other places alongthe vehicle 100. For instance, arrow 110 indicates that a sensor unit(112 in FIG. 1B) may be positioned along the rear of the vehicle 100,such as on or adjacent to the bumper. And arrow 114 indicates a seriesof sensor units 116 arranged along a forward-facing direction of thevehicle. In some examples, the passenger vehicle 100 also may includevarious sensors for obtaining information about the vehicle's interiorspaces (not shown).

FIGS. 1C-D illustrate an example cargo vehicle 150, such as atractor-trailer truck. The truck may include, e.g., a single, double ortriple trailer, or may be another medium or heavy duty truck such as incommercial weight classes 4 through 8. As shown, the truck includes atractor unit 152 and a single cargo unit or trailer 154. The trailer 154may be fully enclosed, open such as a flat bed, or partially opendepending on the type of cargo to be transported. In this example, thetractor unit 152 includes the engine and steering systems (not shown)and a cab 156 for a driver and any passengers. In a fully autonomousarrangement, the cab 156 may not be equipped with seats or manualdriving components, since no person may be necessary.

The trailer 154 includes a hitching point, known as a kingpin, 158. Thekingpin 158 is typically formed as a solid steel shaft, which isconfigured to pivotally attach to the tractor unit 152. In particular,the kingpin 158 attaches to a trailer coupling 160, known as afifth-wheel, that is mounted rearward of the cab. For a double or tripletractor-trailer, the second and/or third trailers may have simple hitchconnections to the leading trailer. Or, alternatively, each trailer mayhave its own kingpin. In this case, at least the first and secondtrailers could include a fifth-wheel type structure arranged to coupleto the next trailer.

As shown, the tractor may have one or more sensor units 162, 164disposed therealong. For instance, one or more sensor units 162 may bedisposed on a roof or top portion of the cab 156, and one or more sidesensor units 164 may be disposed on left and/or right sides of the cab156. Sensor units may also be located along other regions of the cab156, such as along the front bumper or hood area, in the rear of thecab, adjacent to the fifth-wheel, underneath the chassis, etc. Thetrailer 154 may also have one or more sensor units 166 disposedtherealong, for instance along a side panel, front, rear, roof and/orundercarriage of the trailer 154.

By way of example, each sensor unit may include one or more sensors,such as lidar, radar, camera (e.g., optical or infrared), acoustical(e.g., microphone or sonar-type sensor), inertial (e.g., accelerometer,gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPSsensors). While certain aspects of the disclosure may be particularlyuseful in connection with specific types of vehicles, the vehicle may beany type of vehicle including, but not limited to, cars, trucks,motorcycles, buses, recreational vehicles, etc.

There are different degrees of autonomy that may occur for a vehicleoperating in a partially or fully autonomous driving mode. The U.S.National Highway Traffic Safety Administration and the Society ofAutomotive Engineers have identified different levels to indicate howmuch, or how little, the vehicle controls the driving. For instance,Level 0 has no automation and the driver makes all driving-relateddecisions. The lowest semi-autonomous mode, Level 1, includes some driveassistance such as cruise control. Level 2 has partial automation ofcertain driving operations, while Level 3 involves conditionalautomation that can enable a person in the driver's seat to take controlas warranted. In contrast, Level 4 is a high automation level where thevehicle is able to drive without assistance in select conditions. AndLevel 5 is a fully autonomous mode in which the vehicle is able to drivewithout assistance in all situations. The architectures, components,systems and methods described herein can function in any of the semi orfully-autonomous modes, e.g., Levels 1-5, which are referred to hereinas autonomous driving modes. Thus, reference to an autonomous drivingmode includes both partial and full autonomy.

FIG. 2 illustrates a block diagram 200 with various components andsystems of an exemplary vehicle, such as passenger vehicle 100, tooperate in an autonomous driving mode. As shown, the block diagram 200includes one or more computing devices 202, such as computing devicescontaining one or more processors 204, memory 206 and other componentstypically present in general purpose computing devices. The memory 206stores information accessible by the one or more processors 204,including instructions 208 and data 210 that may be executed orotherwise used by the processor(s) 204. The computing system may controloverall operation of the vehicle when operating in an autonomous drivingmode.

The memory 206 stores information accessible by the processors 204,including instructions 208 and data 210 that may be executed orotherwise used by the processors 204. The memory 206 may be of any typecapable of storing information accessible by the processor, including acomputing device-readable medium. The memory is a non-transitory mediumsuch as a hard-drive, memory card, optical disk, solid-state, etc.Systems may include different combinations of the foregoing, wherebydifferent portions of the instructions and data are stored on differenttypes of media.

The instructions 208 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions”, “modules” and “programs” may be usedinterchangeably herein. The instructions may be stored in object codeformat for direct processing by the processor, or in any other computingdevice language including scripts or collections of independent sourcecode modules that are interpreted on demand or compiled in advance. Thedata 210 may be retrieved, stored or modified by one or more processors204 in accordance with the instructions 208. In one example, some or allof the memory 206 may be an event data recorder or other secure datastorage system configured to store vehicle diagnostics and/or detectedsensor data, which may be on board the vehicle or remote, depending onthe implementation.

The processors 204 may be any conventional processors, such ascommercially available CPUs. Alternatively, each processor may be adedicated device such as an ASIC or other hardware-based processor.Although FIG. 2 functionally illustrates the processors, memory, andother elements of computing devices 202 as being within the same block,such devices may actually include multiple processors, computingdevices, or memories that may or may not be stored within the samephysical housing. Similarly, the memory 206 may be a hard drive or otherstorage media located in a housing different from that of theprocessor(s) 204. Accordingly, references to a processor or computingdevice will be understood to include references to a collection ofprocessors or computing devices or memories that may or may not operatein parallel.

In one example, the computing devices 202 may form an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may capable of communicating with various components ofthe vehicle. For example, the computing devices 202 may be incommunication with various systems of the vehicle, including a drivingsystem including a deceleration system 212 (for controlling braking ofthe vehicle), acceleration system 214 (for controlling acceleration ofthe vehicle), steering system 216 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 218 (forcontrolling turn signals), navigation system 220 (for navigating thevehicle to a location or around objects) and a positioning system 222(for determining the position of the vehicle, e.g., including thevehicle's pose). The autonomous driving computing system may employ aplanner module 223, in accordance with the navigation system 220, thepositioning system 222 and/or other components of the system, e.g., fordetermining a route from a starting point to a destination or for makingmodifications to various driving aspects in view of current or expectedtraction conditions.

The computing devices 202 are also operatively coupled to a perceptionsystem 224 (for detecting objects and conditions in the vehicle'senvironment), a power system 226 (for example, a battery and/or gas ordiesel powered engine) and a transmission system 230 in order to controlthe movement, speed, etc., of the vehicle in accordance with theinstructions 208 of memory 206 in an autonomous driving mode which doesnot require or need continuous or periodic input from a passenger of thevehicle. Some or all of the wheels/tires 228 are coupled to thetransmission system 230, and the computing devices 202 may be able toreceive information about tire pressure, balance and other factors thatmay impact driving in an autonomous mode.

The computing devices 202 may control the direction and speed of thevehicle, e.g., via the planner module 223, by controlling variouscomponents. By way of example, computing devices 202 may navigate thevehicle to a destination location completely autonomously using datafrom the map information and navigation system 220. Computing devices202 may use the positioning system 222 to determine the vehicle'slocation and the perception system 224 to detect and respond to objectswhen needed to reach the location safely. In order to do so, computingdevices 202 may cause the vehicle to accelerate (e.g., by increasingfuel or other energy provided to the engine by acceleration system 214),decelerate (e.g., by decreasing the fuel supplied to the engine,changing gears, and/or by applying brakes by deceleration system 212),change direction (e.g., by turning the front or other wheels of vehicle100 by steering system 216 to the left or to the right), and signal suchchanges (e.g., by lighting turn signals of signaling system 218). Thus,the acceleration system 214 and deceleration system 212 may be a part ofa drivetrain or other type of transmission system 230 that includesvarious components between an engine of the vehicle and the wheels ofthe vehicle. Again, by controlling these systems, computing devices 202may also control the transmission system 230 of the vehicle in order tomaneuver the vehicle autonomously.

Navigation system 220 may be used by computing devices 202 in order todetermine and follow a route to a location. In this regard, thenavigation system 220 and/or memory 206 may store map information, e.g.,highly detailed maps that computing devices 202 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways (e.g., including dips, angles, etc.), lanemarkers, intersections, crosswalks, speed limits, traffic signal lights,buildings, signs, real time traffic information, vegetation, or othersuch objects and information. The lane markers may include features suchas solid or broken double or single lane lines, solid or broken lanelines, reflectors, etc. A given lane may be associated with left and/orright lane lines or other lane markers that define the boundary of thelane. Thus, most lanes may be bounded by a left edge of one lane lineand a right edge of another lane line.

The perception system 224 includes sensors 232 for detecting objects andenvironmental factors external to the vehicle. The detected objects maybe other vehicles, obstacles in the roadway, traffic signals, signs,trees, etc. The sensors may 232 may also detect certain aspects ofweather conditions, such as snow, rain or water spray, or puddles, iceor other materials on the roadway. A selected vehicle may includeenhanced sensors to provide water measurements for a roadway segment. Byway of example only, a road weather information sensor from Lufft may beemployed.

By way of example only, the perception system 224 may include one ormore light detection and ranging (lidar) sensors and/or LED emitters,radar units, cameras (e.g., optical imaging devices, with or without aneutral-density filter (ND) filter), positioning sensors (e.g.,gyroscopes, accelerometers and/or other inertial components), infraredsensors, acoustical sensors (e.g., microphones or sonar transducers),and/or any other detection devices that record data which may beprocessed by computing devices 202. Such sensors of the perceptionsystem 224 may detect objects outside of the vehicle and theircharacteristics such as location, orientation, size, shape, type (forinstance, vehicle, pedestrian, bicyclist, etc.), heading, speed ofmovement relative to the vehicle, etc. Ambient conditions (e.g.,temperature and humidity) and roadway conditions such as surfacetemperature, dew point and/or relative humidity, water film thickness,precipitation type, etc. may also be detected by one or more types ofthese sensors.

The perception system 224 may also include other sensors within thevehicle to detect objects and conditions within the vehicle, such as inthe passenger compartment. For instance, such sensors may detect, e.g.,one or more persons, pets, packages, etc., as well as conditions withinand/or outside the vehicle such as temperature, humidity, etc. Stillfurther sensors 232 of the perception system 224 may measure the rate ofrotation of the wheels 228, an amount or a type of braking by thedeceleration system 312, and other factors associated with the equipmentof the vehicle itself.

The raw data from the sensors, including the roadway condition sensors,and the aforementioned characteristics can be processed by theperception system 224 and/or sent for further processing to thecomputing devices 202 periodically or continuously as the data isgenerated by the perception system 224. Computing devices 202 may usethe positioning system 222 to determine the vehicle's location andperception system 224 to detect and respond to objects and roadwayconditions when needed to reach the location safely, e.g., viaadjustments made by planner module 223. In addition, the computingdevices 202 may perform calibration of individual sensors, all sensorsin a particular sensor assembly, or between sensors in different sensorassemblies or other physical housings.

As illustrated in FIGS. 1A-B, certain sensors of the perception system224 may be incorporated into one or more exterior sensor assemblies orhousings. In one example, these may be integrated into the side-viewmirrors on the vehicle. In another example, other sensors may be part ofthe roof-top housing 102, or other sensor housings or units 104, 106a,b, 108 a,b, 112 and/or 116. The computing devices 202 may communicatewith the sensor assemblies located on or otherwise distributed along thevehicle. Each assembly may have one or more types of sensors such asthose described above.

Returning to FIG. 2 , computing devices 202 may include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as a user interfacesubsystem 234. The user interface subsystem 234 may include one or moreuser inputs 236 (e.g., a mouse, keyboard, touch screen and/ormicrophone) and one or more display devices 238 (e.g., a monitor havinga screen or any other electrical device that is operable to displayinformation). In this regard, an internal electronic display may belocated within a cabin of the vehicle (not shown) and may be used bycomputing devices 202 to provide information to passengers within thevehicle. Other output devices, such as speaker(s) 240 may also belocated within the passenger vehicle.

The passenger vehicle also includes a communication system 242. Forinstance, the communication system 242 may also include one or morewireless configurations to facilitate communication with other computingdevices, such as passenger computing devices within the vehicle,computing devices external to the vehicle such as in another nearbyvehicle on the roadway, and/or a remote server system. The networkconnections may include short range communication protocols such asBluetooth™, Bluetooth™ low energy (LE), cellular connections, as well asvarious configurations and protocols including the Internet, World WideWeb, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing.

FIG. 3A illustrates a block diagram 300 with various components andsystems of a vehicle, e.g., vehicle 150 of FIG. 1C. By way of example,the vehicle may be a truck, bus, farm equipment, construction equipment,emergency vehicle or the like, configured to operate in one or moreautonomous modes of operation. As shown in the block diagram 300, thevehicle includes a control system of one or more computing devices, suchas computing devices 302 containing one or more processors 304, memory306 and other components similar or equivalent to components 202, 204and 206 discussed above with regard to FIG. 2 . The control system mayconstitute an electronic control unit (ECU) of a tractor unit of a cargovehicle. As with instructions 208, the instructions 308 may be any setof instructions to be executed directly (such as machine code) orindirectly (such as scripts) by the processor. Similarly, the data 310may be retrieved, stored or modified by one or more processors 304 inaccordance with the instructions 308.

In one example, the computing devices 302 may form an autonomous drivingcomputing system incorporated into vehicle 150. Similar to thearrangement discussed above regarding FIG. 2 , the autonomous drivingcomputing system of block diagram 300 may capable of communicating withvarious components of the vehicle in order to perform route planning anddriving operations. For example, the computing devices 302 may be incommunication with various systems of the vehicle, such as a drivingsystem including a deceleration system 312, acceleration system 314,steering system 316, signaling system 318, navigation system 320 and apositioning system 322, each of which may function as discussed aboveregarding FIG. 2 .

The computing devices 302 are also operatively coupled to a perceptionsystem 324, a power system 326 and a transmission system 330. Some orall of the wheels/tires 228 are coupled to the transmission system 230,and the computing devices 202 may be able to receive information abouttire pressure, balance, rotation rate and other factors that may impactdriving in an autonomous mode. As with computing devices 202, thecomputing devices 302 may control the direction and speed of the vehicleby controlling various components. By way of example, computing devices302 may navigate the vehicle to a destination location completelyautonomously using data from the map information and navigation system320. Computing devices 302 may employ a planner module 323, inconjunction with the positioning system 322, the perception system 324and other subsystems to detect and respond to objects when needed toreach the location safely, similar to the manner described above forFIG. 2 .

Similar to perception system 224, the perception system 324 alsoincludes one or more sensors or other components such as those describedabove for detecting objects and environmental condition (includingroadway conditions) external to the vehicle, objects or conditionsinternal to the vehicle, and/or operation of certain vehicle equipmentsuch as the wheels and deceleration system 312. For instance, asindicated in FIG. 3A the perception system 324 includes one or moresensor assemblies 332. Each sensor assembly 232 includes one or moresensors. In one example, the sensor assemblies 332 may be arranged assensor towers integrated into the side-view mirrors on the truck, farmequipment, construction equipment or the like. Sensor assemblies 332 mayalso be positioned at different locations on the tractor unit 152 or onthe trailer 154, as noted above with regard to FIGS. 1C-D. The computingdevices 302 may communicate with the sensor assemblies located on boththe tractor unit 152 and the trailer 154. Each assembly may have one ormore types of sensors such as those described above.

Also shown in FIG. 3A is a coupling system 334 for connectivity betweenthe tractor unit and the trailer. The coupling system 334 may includeone or more power and/or pneumatic connections (not shown), and afifth-wheel 336 at the tractor unit for connection to the kingpin at thetrailer. A communication system 338, equivalent to communication system242, is also shown as part of vehicle system 300.

FIG. 3B illustrates an example block diagram 340 of systems of thetrailer, such as trailer 154 of FIGS. 1C-D. As shown, the systemincludes an ECU 342 of one or more computing devices, such as computingdevices containing one or more processors 344, memory 346 and othercomponents typically present in general purpose computing devices. Thememory 346 stores information accessible by the one or more processors344, including instructions 348 and data 350 that may be executed orotherwise used by the processor(s) 344. The descriptions of theprocessors, memory, instructions and data from FIGS. 2 and 3A apply tothese elements of FIG. 3B.

The ECU 342 is configured to receive information and control signalsfrom the trailer unit. The on-board processors 344 of the ECU 342 maycommunicate with various systems of the trailer, including adeceleration system 352, signaling system 254, and a positioning system356. The ECU 342 may also be operatively coupled to a perception system358 with one or more sensors for detecting objects and/or conditions inthe trailer's environment and a power system 260 (for example, a batterypower supply) to provide power to local components. Some or all of thewheels/tires 362 of the trailer may be coupled to the decelerationsystem 352, and the processors 344 may be able to receive informationabout tire pressure, balance, wheel speed and other factors that mayimpact driving in an autonomous mode, and to relay that information tothe processing system of the tractor unit. The deceleration system 352,signaling system 354, positioning system 356, perception system 358,power system 360 and wheels/tires 362 may operate in a manner such asdescribed above with regard to FIGS. 2 and 3A.

The trailer also includes a set of landing gear 366, as well as acoupling system 368. The landing gear provides a support structure forthe trailer when decoupled from the tractor unit. The coupling system368, which may be a part of coupling system 334, provides connectivitybetween the trailer and the tractor unit. Thus, the coupling system 368may include a connection section 370 (e.g., for power and/or pneumaticlinks). The coupling system also includes a kingpin 372 configured forconnectivity with the fifth-wheel of the tractor unit.

Example Implementations

In view of the structures and configurations described above andillustrated in the figures, various aspects will now be described inaccordance with aspects of the technology.

While models for road surface and other conditions may be trained onhuman-labeled data, such an approach is subjective and can beerror-ridden. Thus, selected sensor data is employed as a ground truthto the model. Various model architectures can be employed, for instanceusing a Neural Architecture Search (NAS) type model. Different modelarchitectures can be used depending on the type(s) of data, such ason-board lidar data and road graph information. Thus, any DL model thatcan be used to classify/regress road wetness using on-board sensorsignals and other available prior information (such as road graph data,etc.), may be employed.

Various sensors may be located at different places around the vehicle(see FIGS. 1A-D) to gather data from different parts of the externalenvironment. Certain sensors may have different fields of view dependingon their placement around the vehicle and the type of information theyare designed to gather. For instance, different sensors may be used fornear (short range) detection of objects or conditions adjacent to thevehicle (e.g., less than 2-10 meters), while others may be used for far(long range) detection of objects a hundred meters (or more or less) infront of the vehicle. Mid-range sensors may also be employed. Multiplesensor units such as lidars and radars may be positioned toward thefront or rear of the vehicle for long-range object detection. Andcameras and other image sensors may be arranged to provide goodvisibility around the vehicle. Depending on the configuration, certaintypes of sensors may include multiple individual sensors withoverlapping fields of view. Alternatively, other sensors may provideredundant 360° fields of view.

FIG. 4 illustrates a scenario 400 in which a vehicle uses one or moresensors to detect the presence of water along the roadway in order toobtain ground truth data. For instance, the ground truth input mayinclude measurements of the water thickness, e.g., water film thicknessand/or ice coverage on road surfaces. This can be done at a verygranular level, e.g., measuring the thickness on the order of microns.In this scenario, the vehicle may be configured to operate in anautonomous driving mode (or a manual mode), that includes varioussensors at different locations along the exterior of the vehicle. Thiscan include front and/or rear sensor units 402, and a roof-based sensorunit 404, each which may include lidar, radar, optical cameras, acousticsensors and/or other sensors. These or other sensor units may be used tocollect signals of the environment around the autonomous vehicle.

By way of example, the ground truth can be collected using sensors(e.g., front and/or rear sensors 402) designed for water thickness,e.g., water film thickness measurement and/or ice coverage. This couldinclude, e.g., a road weather information sensor from Lufft. Forinstance, the front sensor may obtain data from scans shown via dashedlines 406 _(F), while the rear sensor may obtain data from scans shownvia dashed lines 406 _(R). The roof-based sensor assembly may obtaininformation about objects or conditions around the vehicle as shown bydash-dot lines 408. Notice that the sensors used to collect ground truthdata may only be placed in selected vehicles for the training of deeplearning models during the development phase. After deployment of suchmodels on-board of the autonomous vehicles, these sensors that measurethe road wetness does not need to be installed on vehicles.

The placement of the ground truth collecting sensor(s) around thevehicle may vary depending on the type of vehicle (e.g., sedan, truck,motorcycle, etc.) and other factors, so long as the sensor has a directline of sight to the relevant portion of the roadway. Spray from tiresor other vehicles could potentially have some effect, so to mitigatethis the ground truth sensor should be covered by a protective housing.Also, water droplets passing across the sensor's sensing track canimpact the optical sensing and affect the measurement. However, byavoiding mounting the sensor right above the tire tracks, the likelihoodof water spray flying across the sensing track is small.

Besides sensors used for ground truth, FIG. 5 provides one example 500of sensor fields of view relating to the sensors illustrated in FIG. 1B.Here, should the roof-top housing 102 include a lidar sensor as well asvarious cameras, radar units, infrared and/or acoustical sensors, eachof those sensors may have a different field of view. Thus, as shown, thelidar sensor may provide a 360° FOV 502, while cameras arranged withinthe housing 102 may have individual FOVs 504. A sensor within housing104 at the front end of the vehicle has a forward facing FOV 506, whilea sensor within housing 112 at the rear end has a rearward facing FOV508. The housings 106 a, 106 b on the driver's and passenger's sides ofthe vehicle may each incorporate lidar, radar, camera and/or othersensors. For instance, lidars within housings 106 a and 106 b may have arespective FOV 510 a or 510 b, while radar units or other sensors withinhousings 106 a and 106 b may have a respective FOV 511 a or 511 b.Similarly, sensors within housings 108 a, 108 b located towards the rearroof portion of the vehicle each have a respective FOV. For instance,lidars within housings 108 a and 108 b may have a respective FOV 512 aor 512 b, while radar units or other sensors within housings 108 a and108 b may have a respective FOV 513 a or 513 b. And the series of sensorunits 116 arranged along a forward-facing direction of the vehicle mayhave respective FOVs 514, 516 and 518. Each of these fields of view ismerely exemplary and not to scale in terms of coverage range.

Examples of lidar, camera and radar sensors and their fields of view fora cargo-type vehicle (e.g., vehicle 150 of FIGS. 1C-D) are shown inFIGS. 6A and 6B. In example 600 of FIG. 6A, one or more lidar units maybe located in rooftop sensor housing 602, with other lidar units in sidesensor housings 604. In particular, the rooftop sensor housing 602 maybe configured to provide a 360° FOV. A pair of sensor housings 604 maybe located on either side of the tractor unit cab, for instanceintegrated into a side view mirror assembly or along a side door orquarter panel of the cab. In one scenario, long range lidars may belocated along a top or upper area of the sensor housings 602 and 604.The long range lidar may be configured to see over the hood of thevehicle. And short range lidars may be located in other portions of thesensor housings 602 and 604. The short range lidars may be used by theperception system to determine whether an object such as anothervehicle, pedestrian, bicyclist, etc. is next to the front or side of thevehicle and take that information into account when determining how todrive or turn. Both types of lidars may be co-located in the housing,for instance aligned along a common vertical axis.

As illustrated in FIG. 6A, the lidar(s) in the rooftop sensor housing602 may have a FOV 606. Here, as shown by region 608, the trailer orother articulating portion of the vehicle may provide signal returns,and may partially or fully block a rearward view of the externalenvironment. Long range lidars on the left and right sides of thetractor unit have FOV 610. These can encompass significant areas alongthe sides and front of the vehicle. As shown, there may be an overlapregion 612 of their fields of view in front of the vehicle. The overlapregion 612 provides the perception system with additional or informationabout a very important region that is directly in front of the tractorunit. This redundancy also has a safety aspect. Should one of the longrange lidar sensors suffer degradation in performance, the redundancywould still allow for operation in an autonomous mode. Short rangelidars on the left and right sides have smaller FOV 614. A space isshown between different fields of view for clarity in the drawing;however in actuality there may be no break in the coverage. The specificplacements of the sensor assemblies and fields of view is merelyexemplary, and may differ depending on, e.g., the type of vehicle, thesize of the vehicle, FOV requirements, etc.

FIG. 6B illustrates an example configuration 620 for either (or both) ofradar and camera sensors in a rooftop housing and on both sides of atractor-trailer, such as vehicle 150 of FIGS. 1C-D. Here, there may bemultiple radar and/or camera sensors in each of the sensor housings 602and 604 of FIG. 6A. As shown, there may be sensors in the rooftophousing with front FOV 622, side FOV 624 and rear FOV 626. As withregion 608, the trailer may impact the ability of the sensor to detectobjects behind the vehicle. Sensors in the sensor housings 604 may haveforward facing FOV 628 (and side and/or rear fields of view as well). Aswith the lidars discussed above with respect to FIG. 6A, the sensors ofFIG. 6B may be arranged so that the adjoining fields of view overlap,such as shown by overlapping region 630. The overlap regions heresimilarly can provide redundancy and have the same benefits should onesensor suffer degradation in performance.

Example Scenarios

As shown in example 700 of FIG. 7 , a processing system 702 may receivevarious inputs from vehicles and other sources. For instance, on-boardsignals received from a passenger vehicle 704 a or a truck 704 b caninclude lidar returns, camera images/on-board video, radar returns,audio signals, and ground truth via a road wetness sensor output (e.g.,from a sensor configured to detect road weather information includingwater film height, ice percentage, etc. via optical spectroscopy orother technique). In addition, the output from other perceptionmodules/models of the vehicle (e.g., puddle detectors and filteringmodules), may also be part of the on-board signals.

Off-board signals provided by external sources 706 (e.g., 706 a and 706b) can include, by way of example, weather station information, publicweather forecasts, road graph data, human-labeled road wetness groundtruth examples, crowdsourced information, and observations from othervehicles (e.g., as part of a fleet of vehicles) in nearby locations togive additional context about the road wetness.

Example 800 of FIG. 8 illustrates such on-board factors 802 and offboardfactors 804, which can be gathered via a network 708 and stored astraining inputs 710 as shown in FIG. 7 . Here, for instance, the weatherstation information and public weather forecasts may come from a thirdparty source(s) or external system 706 a. The road graph data,human-labeled wetness ground truth examples, observations from othervehicles, etc., may come from system 706 b.

As shown, the processing system 702 includes one or more processors 712,memory 714 having instructions 716 and data 718, as well optional userinputs 720 and a display 722. Each of these may be configured andoperate in a manner equivalent to what is described above with regard tothe computing devices and processing systems of FIGS. 2 and 3A-B. Thedata 718 may include one or more models 724, such as the DL modelsdescribed herein.

Some or all of these signals may be fused together. For example, machineleaning can handle fusion from different sources. Machine learning takesinput from multiple sensors and builds a model to output the finalresults. In this modeling process all the inputs (or a selected subsetof the inputs) are fused together. This can be done by creating specialembedding layers in the model that combine input in a human-engineeredway, or directly build an end-to-end architecture that takes all inputdirectly into the model. The embedding layers can be human engineered,or also the embeddings can be learned. The sensor data and embeddingscan be combined anywhere in the model, at the very beginning as rawdata, later as embeddings, or somewhere in between.

Different signals may be given different weights. For instance, one canconstruct human-engineered features from raw sensor inputs, where some apriori knowledge regarding which sensor should be emphasized can beencoded into the construction of a feature. By way of example, thesystem may aggregate the lidar data in an area into a single value to beused as the input in the model but gives different weights to points atdifferent places in the area when constructing this value. Anotherapproach is to utilize the learning capability of the deep net andinclude the weights of different input into model parameters. Then theweights of different input can be learned in the model training process.

In one scenario, the models learn the embeddings and the weights on eachone. There are two kinds of weights. First, in the input, differentinputs may be weighted differently. This can be done with embeddinglayers that are human engineered (e.g., selected by a system engineer),or simply figured out by the model itself when it trains and convergesto different weights for different input channels. These weights maygenerally be the same for all examples.

The second kind is the weight that can be assigned to differentexamples, quantifying how important they are for evaluation of modelquality. For example, in classification model, the examples with groundtruth water film height very close to the threshold of dry/wet areassigned with less weight, because it is more likely that thebinarization into dry/wet of such examples are ambiguous and/or theground truth from such examples may be corrupted with measurement noise.

The road may have a continuum of conditions from wet to dry. The systemmay seek to identify regions of the roadway that are wet, regions thatare dry, and potentially ambiguous areas in between. For instance,example 900 of FIG. 9 illustrates that a portion 902 of the rightmostlane is wet. This may be due to a puddle or accumulation of water thatis, e.g., 1.0-4.0 mm deep (or more). The dry region 904 may have nowater accumulation (e.g., less than 0.03 mm). And there may be an area906 between the wet and dry regions that may have some wateraccumulation (e.g., a water film of between 0.02-2.0 mm), where it maybe ambiguous as to whether this should be classified as “wet” or “dry”.In one scenario, the information for the ambiguous region may be givenless weight, as indicated above for the second kind of weighting.

Road wetness values can indicate a probability of whether that portionof the roadway is wet at all, or how wet it is. For classificationmodels, the output of the model is not simply some classes (e.g., “wet”or “dry”), but a probability that some road region falling into certainclasses. The probability can indicate how confident the system is withthe classification results, and also if there are any alternativepotential classes with lower probability. Thus, for the example of FIG.9 , the region 906 may have a higher probability (e.g., 60-90%) of being“wet”, and a lower probability (e.g., 10-30%) of being “dry”.

The model output can have different granularity. By way of example, forclassification models there could be only two classes such as dry/wet,or more classes based on water film height, such as one class for eachincrement of certain water film height (e.g., each 0.25, 0.5 or 1.0 mm).There could even be a regression model that provides continuousestimation of water film height on the road. The granularity may bedecided based on needs and requirements when making (autonomous) drivingdecisions. By way of example, granularity may be useful when decidingwhether to drive through or avoid a particular section of the (wet)roadway.

While wet and dry are two outputs of the model, additional granularitycan include, by way of example only, “slightly wet” (e.g., damp) wherethere is some amount of moisture on the road surface below a thresholdfor “wet”; “icy” where the water is substantially in the form of ice(e.g., a percentage of ice crystals in a sample exceeds a threshold);“snow” where the water is in the form of small white ice crystals thatcovers a selected portion of the roadway; “chemically wet”, e.g., wherethe water molecules have not turned to ice due to a de-icing chemical onthe roadway; and/or “other”, for instance where the specific nature ofthe road condition does not fall into any other category.

Statistical analysis may be employed before building the DL models,e.g., to discover which on-board signals correlate most effectively withthe ground truth, and also to eliminate or deemphasize any on-boardsignals or external parameters that do not have good correlation withthe ground truth. Relevant statistical parameters include mean andstandard deviation values for the sensor data. The sensor signal returnscan be bucketed based on different conditions (e.g., distance from theself-driving vehicle). This helps determine the useful range and toeliminate conditions that do not matter or otherwise affect thestatistics. For instance, road materials, road wear or surface type(e.g., grooved) may not be relevant, and off-road returns excluded.Temperature, light conditions and other ambient factors may or may notbe relevant.

Another factor can include identifying the placement/positioning of theon-board sensors that provide the most useful information. During atesting phase, the sensors may be placed at different locations alongthe vehicle to see which one gives stronger signals (e.g., signals thatmore closely correlate with the measured road wetness ground truth.

The result of this analysis is a highly useful subset of data, whichmasks out returns from dynamic objects on the roadway to avoid the noiseintroduced by vehicles, pedestrians, bicyclists and other road users.For instance, lidar sensor information may include intensity andreflectivity, and the statistical evaluation may show that intensity ismore relevant than reflectivity. Thus, a strong signal input may belaser data that is limited in range and height. By way of example only,the range of the laser points that yields the most difference betweenwet and dry road surfaces may be on the order of 30-50 m from thevehicle, and the threshold to separate wet and dry road surfaces basedon measured water film thickness may be on the order of 5-20 μm. Inaddition, the reflection of light on water impacts the return intensitybecause water changes how much light gets reflected back to and awayfrom the sensor. This is the primary signal. Height gives geometryinformation and helps determine which point is from the road. Elongationand secondary return give additional information regarding thereflection surface. Another useful input is road graph data from a map,which gives information of what points are on road or off road.

The probability of road wetness is the output of the classificationmodel. To obtain a dry/wet classification, a threshold on theprobability is given. An example is using probability of wet=0.5 as thethreshold. During the training of the model, this classification iscompared to the ground truth as an evaluation of the quality of thecurrent model, and the model parameters are adjusted accordingly.

The model structure is a deep net, where the exact structure andparameters can be searched through automated machine learning. This maybe done by a method of automated machine learning such as NAS, which isa technique for automating the design of artificial neural networksinstead of human designed architecture. According to one aspect of thetechnology, automated machine learning techniques are used to optimizethe design of the model. Examples of automated model selection includevariants of NAS (such as TuNAS), automated hyperparameter optimization(such as Vizier), and automated data augmentation. This way, thedocument can achieve a better understanding in the general machinelearning audience. An example process would be to give a set of basicmodel architecture elements (such as some representative layers) and usereinforcement learning to search for the best combination among theseelements.

Model accuracy can be improved in different ways. This can includesmoothing the measurements from the road wetness sensor(s) to obtain amore robust estimation of ground truth, balancing wet and dry examplesin training dataset to avoid models with skewed performance, anddesigning a loss function that gives more emphasis to the examples ofhigher confidence to be wet or dry. The system may use a low pass filterto filter out high frequency noise.

The loss function can be the weighted sum, among all training examples,of the square of the difference between the ground truth and the modeloutput. In the weighted sum, higher weights are assigned to exampleswith higher confidence while the examples with lower confidence getslower weights.

FIG. 10 illustrates an example 1000 of the road condition deep learningmodel architecture in accordance with aspects of the technology. Thearchitecture may be implemented via the processing system of FIG. 7 . Asshown in block 1002, both signals 1002 _(A) from onboard sensors andoffboard signals 1002 _(B) are inputs to the system (e.g., traininginputs 710 of FIG. 7 ). These inputs, which may be any or all of thetypes described above, are fed into a feature extraction layer 1004. Thefeature extraction layer takes an initial set of input data and buildsderived features. These features may be of reduced dimensions, may beinformative and non-redundant, may facilitate the subsequent learning,and may lead to better human interpretations. The extracted features areapplied to a pooling layer 1006. The pooling layer can reduce thedimension of data representation, and the number of parameters need tobe learnt in the model, and enable smaller model structure and fasterlearning.

The pooled information output by the pooling layer 1006 is fed into amodule 1008 that includes a convolution layer 1010 and an activationlayer 1012. The convolution layer 1010 transforms input images intoimages of potentially different size and parameters, and thus extractsfeatures that maybe hidden in the input images. The activation layer1012 provides non-linearity to the model through different activationfunctions. Processing within the module 1008 may be repeated multipletimes, as indicated by dash-dot line 1013. Repeating such layers addsdepth to the deep learning models and allow us to learn more complicatedmodel structures. The exact number of repetition (e.g., 2, 3, or moretimes) can be both human-engineered or searched through NAS.

Next, data output from module 1008 is fed to a fully connected layer1014. The fully connected layer integrates outputs from the previouslayer into a vector of desired size. This may capture the complicatedrelationship among high-level features. Output 1016 is, e.g., theclassification or the continuous estimation of the road wetness. Thus,the various layers form the road wetness model, and the model givesoutput 1016 such as classification or estimation. While individuallayers 1004, 1006, 1010, 1012 and 1014 are shown in example 1000 of FIG.10 , there can be one or more such layers for each of featureextraction, pooling, convolution, activation, and fully connected. It isalso possible that one or more of these layers are not present in themodel. For instance, in some scenarios the pooling, convolution and/oractivation layers may be omitted.

The end result of this modeling approach is the ability to give adiscrete classification or continuous regression/estimation of roadwetness, which has a number of beneficial uses. These include thetriggering of safety precautions (e.g., pulling over for roads too wetto handle); causing a change in real-time motion control (e.g.,adjusting acceleration/deceleration, braking distance, changing lanes,etc.); making changes to the perception system (e.g., modifyingthresholds for filtering, sensor noise level, sensor field of viewadaptation, sensor validation logic, pedestrian detectors, etc.);affecting how the wiper system (or any sensor cleaning system) operates;changing models for predicting behavior of other road users (e.g., othervehicles might drive slower, pedestrians or bicyclists might moveerratically to avoid rain/puddles, etc.); and changing planner behavior(such as where to pick up or drop off, selecting alternative routes orlanes of travel, etc.). Such information may be provided to vehiclesacross a fleet of vehicles, such as part of a general system update orbased on current or projected weather conditions to assist schedulingand routing of the fleet.

For instance, FIG. 11A illustrates a first scenario 1100, in which atruck 1102 runs over a wet region 1104 of a roadway. As shown, thiscauses a spray of water 1106 from the truck's tires. In this scenario,car 1108 may determine that there will be the spray of water based onthe road conditions (e.g., depths of the water film on the roadway).Thus, in response to this determination, the car 1108 may make anadjustment to the driving path as shown by dotted line 1110, in view ofother objects along the roadway such as vehicle 1112.

FIG. 11B illustrates a second scenario 1120, in which vehicle 1122observes bicycle 1124 approaching a wet area (e.g., a puddle) 1126.Here, based on information according to the road wetness model and otherfactors (such as an observed object being a bicycle), the vehicle 1122may predict that the bicycle will alter its trajectory to avoid the wetarea as shown by dotted line 1128. As a result, the vehicle 1122 maybrake or cease accelerating to allow the bicycle 1124 sufficient room tomove around the wet area.

As noted above, the technology is applicable for various types ofwheeled vehicles, including passenger cars, buses, motorcycles, RVs,emergency vehicles, and trucks or other cargo carrying vehicles.

In addition to using the road condition model information for operationof the vehicle, this information may also be shared with other vehicles,such as vehicles that are part of a fleet. This can be done to aid inroute planning, gathering of additional ground truth data, modelupdates, etc.

One example of data sharing is shown in FIGS. 12A and 12B. Inparticular, FIGS. 12A and 12B are pictorial and functional diagrams,respectively, of an example system 1200 that includes a plurality ofcomputing devices 1202, 1204, 1206, 1208 and a storage system 1210connected via a network 1216. System 1200 also includes exemplaryvehicles 1212 and 1214, which may be configured the same as or similarlyto vehicles 100 and 150 of FIGS. 1A-B and 1C-D, respectively. Vehicles1212 and/or vehicles 1214 may be part of a fleet of vehicles. Althoughonly a few vehicles and computing devices are depicted for simplicity, atypical system may include significantly more.

As shown in FIG. 12B, each of computing devices 1202, 1204, 1206 and1208 may include one or more processors, memory, data and instructions.Such processors, memories, data and instructions may be configuredsimilarly to the ones described above with regard to FIGS. 2 and 3A-B.

The various computing devices and vehicles may communicate via one ormore networks, such as network 1216. The network 1216, and interveningnodes, may include various configurations and protocols including shortrange communication protocols such as Bluetooth™, Bluetooth LE™, theInternet, World Wide Web, intranets, virtual private networks, wide areanetworks, local networks, private networks using communication protocolsproprietary to one or more companies, Ethernet, WiFi and HTTP, andvarious combinations of the foregoing. Such communication may befacilitated by any device capable of transmitting data to and from othercomputing devices, such as modems and wireless interfaces.

In one example, computing device 1202 may include one or more servercomputing devices having a plurality of computing devices, e.g., a loadbalanced server farm or cloud computing system, that exchangeinformation with different nodes of a network for the purpose ofreceiving, processing and transmitting the data to and from othercomputing devices. For instance, computing device 1202 may include oneor more server computing devices that are capable of communicating withthe computing devices of vehicles 1212 and/or 1214, as well as computingdevices 1204, 1206 and 1208 via the network 1216. For example, vehicles1212 and/or 1214 may be a part of one or more fleets of vehicles thatcan be dispatched by a server computing device to various locations. Inthis regard, the computing device 1202 may function as a dispatchingserver computing system which can be used to dispatch vehicles todifferent locations in order to pick up and drop off passengers and/orto pick up and deliver cargo. In addition, server computing device 1202may use network 1216 to transmit and present information to a user ofone of the other computing devices or a passenger of a vehicle. In thisregard, computing devices 1204, 1206 and 1208 may be considered clientcomputing devices.

As shown in FIG. 12A each client computing device 1204, 1206 and 1208may be a personal computing device intended for use by a respective user1218, and have all of the components normally used in connection with apersonal computing device including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display (e.g., a monitor havinga screen, a touch-screen, a projector, a television, or other devicesuch as a smart watch display that is operable to display information),and user input devices (e.g., a mouse, keyboard, touchscreen ormicrophone). The client computing devices may also include a camera forrecording video streams, speakers, a network interface device, and allof the components used for connecting these elements to one another.

Although the client computing devices may each comprise a full-sizedpersonal computing device, they may alternatively comprise mobilecomputing devices capable of wirelessly exchanging data with a serverover a network such as the Internet. By way of example only, clientcomputing devices 1206 and 1208 may be mobile phones or devices such asa wireless-enabled PDA, a tablet PC, a wearable computing device (e.g.,a smartwatch), or a netbook that is capable of obtaining information viathe Internet or other networks.

In some examples, client computing device 1204 may be a remoteassistance workstation used by an administrator or operator tocommunicate with passengers of dispatched vehicles. Although only asingle remote assistance workstation 1204 is shown in FIGS. 12A-12B, anynumber of such work stations may be included in a given system.Moreover, although operations work station is depicted as a desktop-typecomputer, operations works stations may include various types ofpersonal computing devices such as laptops, netbooks, tablet computers,etc.

Storage system 1210 can be of any type of computerized storage capableof storing information accessible by the server computing devices 1202,such as a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, flash driveand/or tape drive. In addition, storage system 1210 may include adistributed storage system where data is stored on a plurality ofdifferent storage devices which may be physically located at the same ordifferent geographic locations. Storage system 1210 may be connected tothe computing devices via the network 1216 as shown in FIGS. 12A-B,and/or may be directly connected to or incorporated into any of thecomputing devices.

Storage system 1210 may store various types of information. Forinstance, the storage system 1210 may also store autonomous vehiclecontrol software and/or road condition models, which may be used byvehicles, such as vehicles 1212 or 1214, to operate such vehicles in anautonomous driving mode. Storage system 1210 may store map information,route information, weather condition information, road surfaceinformation, vehicle models for the vehicles 1212 and 1214, weatherinformation, etc. This information may be shared with the vehicles 1212and 1214, for instance to help with real-time route planning and drivinganalysis by the on-board computer system(s).

The remote assistance workstation 1204 may access the stored informationand use it to assist operation of a single vehicle or a fleet ofvehicles. By way of example, a lead vehicle may detect a wet condition,such as due to standing water, ice or snow along a road segment, andsend information about the wet condition to the remote assistanceworkstation 1204. In turn, the remote assistance workstation 1204 maydisseminate the information to other vehicles in the fleet, so that theymay alter their routes.

In a situation where there are passengers, the vehicle or remoteassistance may communicate directly or indirectly with the passengers'client computing device. Here, for example, information may be providedto the passengers regarding current driving operations, changes to theroute in response to the situation, etc.

FIG. 13 illustrates an example process 1300 that is a method forgenerating a road condition deep learning model. The method comprisingreceiving at block 1302 as a first set of training inputs, by one ormore processors, sensor data of an environment along a portion of aroadway from one or more on-board vehicle sensors. At block 1304 themethod includes receiving as a second set of training inputs, by the oneor more processors, off-board information associated with the portion ofthe roadway.

At block 1306 the method includes evaluating, by the one or moreprocessors, the received first set of training inputs and the receivedsecond set of training inputs with respect to ground truth data for theportion of the roadway. The ground truth data includes one or moremeasurements of water thickness, e.g., water film thickness or icecoverage across one or more areas of the portion of the roadway to giveclassification or continuous estimation of wetness along the one or moreareas of the portion of the roadway. The evaluating generates roadwetness information based on the received first and second sets oftraining inputs and the ground truth data.

At block 1308 the method also includes generating the road conditiondeep learning model from the road wetness information. And at block 1310the method stores the generated road condition deep learning model inmemory. This can be memory of a back-end system such as storage system1210 of FIG. 12A-B, or memory of a self-driving vehicle such as memory206 of FIG. 2 or memory 306 of FIG. 3A. When stored in memory of aself-driving vehicle, the model can be used during real-time drivingoperations of the vehicle. For instance, the model can be deployed inautonomous vehicles, such as a fleet of vehicles shown in FIG. 12A forclassifying/regressing the road wetness with on-board and/or off-boardsignals as the input, without referring to the ground truth. The modelcan be applied by each vehicle to enhance autonomous operation. This caninclude, for instance, altering current driving actions (e.g., changinglanes, slowing down, changing the rate of deceleration, speeding up,etc.), modifying planned routes or trajectories, activating on-boardcleaning systems (e.g., a wiper system, defogger, defroster or thelike), etc.

Unless otherwise stated, any alternative examples are not mutuallyexclusive, but may be implemented in various combinations to achieveunique advantages. As these and other variations and combinations of thefeatures discussed above can be utilized without departing from thesubject matter defined by the claims, the foregoing description of theembodiments should be taken by way of illustration rather than by way oflimitation of the subject matter defined by the claims. In addition, theprovision of the examples described herein, as well as clauses phrasedas “such as,” “including” and the like, should not be interpreted aslimiting the subject matter of the claims to the specific examples;rather, the examples are intended to illustrate only one of manypossible embodiments. Further, the same reference numbers in differentdrawings can identify the same or similar elements. The processes orother operations may be performed in a different order orsimultaneously, unless expressly indicated otherwise herein.

The invention claimed is:
 1. A system configured to operate a vehicle inan autonomous driving mode, the system comprising: memory storing a roadcondition deep learning model, the model relating to a discreteclassification or continuous regression/estimation of road wetness; andone or more processors operatively coupled to the memory, the one ormore processors being configured to: receive sensor data from one ormore sensors of a perception system of the vehicle while operating inthe autonomous driving mode, the one or more sensors being configured todetect objects or conditions in an environment around the vehicle; usethe stored model to generate information associated with the discreteclassification or continuous regression/estimation of road wetness basedon the received sensor data; and use the generated information tocontrol operation of the vehicle in the autonomous driving mode.
 2. Thesystem of claim 1, wherein the model is formed by evaluating a first setof training inputs of sensor data of an environment along a portion of aroadway from one or more on-board sensors and a second set of traininginputs of off-board information associated with the portion of theroadway with respect to ground truth data for the portion of theroadway, the ground truth data including one or more measurements ofwater thickness across one or more areas of the portion of the roadway.3. The system of claim 2, wherein the first set of training inputs ofsensor data includes one or more of lidar returns, camera still images,camera video, radar returns, audio signals, or output from a vehicleon-board module.
 4. The system of claim 2, wherein the second set oftraining inputs of off-board information includes one or more of weatherstation information, public weather forecasts, road graph data,crowdsourced information, or observations from one or more othervehicles.
 5. The system of claim 1, wherein controlling operation of thevehicle in the autonomous mode using the generated information includesalteration of a current driving action of the vehicle in the autonomousdriving mode.
 6. The system of claim 1, wherein controlling operation ofthe vehicle in the autonomous mode using the generated informationincludes modification of a planned route or trajectory.
 7. The system ofclaim 1, wherein controlling operation of the vehicle in the autonomousmode using the generated information includes activation of an on-boardcleaning system.
 8. The system of claim 1, wherein controlling operationof the vehicle in the autonomous mode using the generated informationincludes triggering one or more safety precautions.
 9. The system ofclaim 1, wherein controlling operation of the vehicle in the autonomousmode using the generated information includes modification of athreshold for one or more aspects of the perception system.
 10. Thesystem of claim 9, wherein modification of a threshold for one or moreaspects of the perception system including at least one of modifying athreshold for filtering, a sensor noise level, a sensor field of viewadaptation, sensor validation logic, or a pedestrian detector.
 11. Thesystem of claim 1, wherein controlling operation of the vehicle in theautonomous mode using the generated information includes changing amodel for predicting behavior of other road users.
 12. A vehicleconfigured to operate in an autonomous driving mode, the vehiclecomprising: a perception system including one or more sensors configuredto detect objects or conditions in an environment around the vehicle;memory storing a road condition deep learning model, the model relatingto a discrete classification or continuous regression/estimation of roadwetness; and one or more processors operatively coupled to the memoryand the perception system, the one or more processors being configuredto: receive sensor data from the one or more sensors of the perceptionsystem of the vehicle while operating in the autonomous driving mode;use the stored model to generate information associated with thediscrete classification or continuous regression/estimation of roadwetness based on the received sensor data; and use the generatedinformation to control operation of the vehicle in the autonomousdriving mode.
 13. The vehicle of claim 12, wherein controlling operationof the vehicle in the autonomous mode using the generated informationincludes alteration of a current driving action of the vehicle in theautonomous driving mode.
 14. The vehicle of claim 12, whereincontrolling operation of the vehicle in the autonomous mode using thegenerated information includes modification of a planned route ortrajectory.
 15. The vehicle of claim 12, wherein controlling operationof the vehicle in the autonomous mode using the generated informationincludes activation of an on-board cleaning system.
 16. The vehicle ofclaim 12, wherein controlling operation of the vehicle in the autonomousmode using the generated information includes triggering one or moresafety precautions.
 17. The vehicle of claim 12, wherein controllingoperation of the vehicle in the autonomous mode using the generatedinformation includes modification of a threshold for one or more aspectsof the perception system.
 18. A method for operating a vehicle in anautonomous driving mode, the method comprising: receiving, by one ormore processors of the vehicle, sensor data from one or more sensors ofa perception system of the vehicle while operating in the autonomousdriving mode, the one or more sensors being configured to detect objectsor conditions in an environment around the vehicle; using, by the one ormore processors, a stored a road condition deep learning model togenerate information associated with a discrete classification orcontinuous regression/estimation of road wetness based on the receivedsensor data; and using, by the one or more processors, the generatedinformation to control operation of the vehicle in the autonomousdriving mode.
 19. The method of claim 18, wherein controlling operationof the vehicle in the autonomous mode using the generated informationincludes at least one of: altering a current driving action of thevehicle in the autonomous driving mode; modifying a planned route ortrajectory for the vehicle; or activating an on-board cleaning system.20. The method of claim 18, wherein controlling operation of the vehiclein the autonomous mode using the generated information includes at leastone of: triggering one or more safety precautions; or modifying athreshold for one or more aspects of the perception system.